Constructive Induction On Decision Trees

Selective induction techniques perform poorly when the features are inappropriate for the target concept. One solution is to have the learning system construct new features automatically ; unfortunately feature construction is a difficult and poorly understood problem. In this paper we present a definition of feature construction in concept learning, and offer a framework for its study based on four aspects: detection, selection, generalization, and evaluation. This framework is used in the analysis of existing learning systems and as the basis for the design of a new system, CITRE. CITRE performs feature construction using decision trees and simple domain knowledge as constructive biases. Initial results on a set of spatial-dependent problems suggest the importance of domain knowledge and feature generalization , i.e., constructive induction. 1 Introduction Good representations are often crucial for solving difficult problems in AI. Finding suitable problem representations , however, can be difficult and time consuming. This is especially true in machine learning: learning can be relatively easy if the training examples are presented in a suitable form, but when the features used in describing examples are inappropriate for the target concept, learning can be difficult or impossible using selective induction methods. To overcome this problem a learning system needs to be capable of generating appropriate features for new situations. This paper is concerned with the automated construction of new features to facilitate concept learning, an issue closely related to the "problem of new terms" [Di-etterich et a/., 1982] and constructive induction [Michal-ski, 1983]. We begin by defining "feature construction" in the context of concept learning from examples, and then proceed to identify four inherent aspects: detection, selection, generalization, and evaluation. These aspects comprise an analytical framework for studying feature construction which we describe through examples drawn from the existing systems of BACON.1, BOGART, DUCE, PLSO, and STAGGER. This framework serves as the basis for the design of CITRE, a system that performs feature construction on decision trees using simple domain knowledge. The results of our initial experiments demonstrate CITRE'S ability to improve learning through feature construction in a tic-tac-toe classification problem. Extensions of CITRE for this and other problems are also discussed. 2 The Problem We state the following definition of feature construction 1 in concept learning: Feature Construction: the application of a set of constructive operators {o 1 , 0 2 , ...o n } to a set of existing features {f 1 ,f 2 …